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When do you do risk analysis ? 


Risk analysis and response planning must be 
done during the initial planning phase of the 
project. Ideally, risk analysis and response 
planning is done during the project proposal 
phase and revisited on a regular basis. 

"70% of a project's cost at completion is committed 
by the time the first 5% of the project's budget is 
actually spent. " 
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The Elements of Risk 

Risk is composed of TWO elements: 

1. ) The UNCERTAINTY (expressed as a probability (Pf) of 
achieving a project performance objective 

AND, 

2. ) The CONSEQUENCES (Cf) of a risk event 

Risk= Pf x Cf 

Caution is needed, of course in using this approach. It is necessary to 
be wary of multiplying 2 pieces of information together to produce a 
figure which may ,make an account's eyes light up but be of little 
practical value to a project manager. 

Risk Assessment Matrix 



Low Medium High 


Probability of Failure 
(1 - Probability of Success) 
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Characterization of Technology Risk 

(utilization for system development) 


■ Probability of failure to: 

- Reach maturity for system integration 
(programmatic failure) 

- And meet Technical Performance Measures 
goals (technical failure) 

■ Impact on overall system performance of 
failing to meet TPM goals 

Measures of 

Probability of Failure 

• The Probability of Failure is measured by the three measures used for 
programs or projects - cost, schedule, and performance. 


Performance (technical failure) 



Cost Schedule 

(programmatic failure) 
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Measures of Programmatic Failure 

• Development difficulty 

- Technology Readiness Level Gap (Initial to TRL6) 

- Research and Development Degree of Difficulty 

- TPM gap 

• Requirements, requirements flowdown, interface requirements, etc. 

• Schedule 

- Defined schedule showing maturity increasing/adequate analysis and 
testing 

- Critical Path 

- Adequate slack 

- High risk items, work around 

- Exit criteria for every milestone 

• Cost 

- Defined cost for all milestones 

- Costs include NASA and contractor 

• Management and technical team (experienced) 


NASA's TECHNOLOGY READINESS LEVEL 
(Scale for Tracking Risk Reduction) 

9 - Actual system "flight proven" on operational flight 

8 - Actual system completed and "flight qualified" through test and demonstration 
7 - System prototype demonstrated in flight 

6 - System/Subsystem (configuration) model or prototype demonstrated/validation 
in a relevant environment 

5 - Component (or breadboard) verification in a relevant environment 

4 - Component and/or breadboard test in a laboratory environment 

3 - Analytical & experimental critical function, or characteristic proof-of-concept, or 
completed design 

2 - Technology concept and/or application formulated (candidate selected) 

1 - Basic principles observed and reported 

( Technology Readiness Level of 6 is usually 
required for Development 
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NASA’s 

Technology Readiness Levels (Software) 


System Test, 
Launch & 
Operations 


System/Subsystem 

Development 


TRL 9 
TRL 8 
TRL 7 


Technology 

Demonstration 


Technology 

Development 


Research to 
Prove Feasibility 


Basic Technology 
Research 



TRL 9: Actual system “mission proven” through successful mission operations 

Thoroughly debugged software readily repeatable. Fully integrated with operational hardware/software 
systems. All documentation completed. Successful operational experience. Sustaining software 
engineering support in place Actual system fully demonstrated. 

TRL 8: Actual system completed and “mission qualified” through test and 
demonstration in an operational environment Thoroughly debugged software. Fully 
integrated with operational hardware and software systems. Most user documentation, training 
documentation, and maintenance documentation completed. All functionality tested in simulated and 
operational scenarios. V&V completed. 

TRL 7: Initial system demonstration in high-fidelity environment (parallel or 
shadow mode operation) Most functionality available for demonstration and test. Well integrated 
with operational hardware/software systems. Most software bugs removed. Limited documentation 
available. 

TRL 6: System/subsystem prototype validated in a relevant end-to-end 
environment Prototype implementations on full scale realistic problems. Partially integrated with 
existing hardware/software systems. Limited documentation available. Engineering feasibility fully 
demonstrated. 

TRL 5: Module and/or subsystem qualified in relevant environment Prototype 
implementations conform to target environment / interfaces. Experiments with realistic problems. 
Simulated interfaces to existing systems. 

TRL 4: Module and/or subsystem qualified in laboratory environment Standalone 
prototype implementations. Experiments with full scale problems or data sets. 

TRL 3: Analytical and experimental critical function and/or characteristic proof- 
of-concept Limited functionality implementations. Experiments with small representative data sets. 
Scientific feasibility fully demonstrated. 

TRL 2: Technology concept and/or application formulated Basic principles coded 
Experiments with synthetic data. Mostly applied research. 

TRL 1: Basic principles observed and reported Basic properties of algorithms, 
representations & concepts. Mathematical formulations. Mix of basic and applied research. 


Measures of Programmatic Failure 

• Development difficulty 

- Technology Readiness Level Gap (Initial to TRL6) 

Research and Development Degree of Difficulty 

- TPM gap 

* Requirements, requirements flowdown, interface requirements, etc. 

* Schedule 

Defined schedule showing maturity increasing/adequate analysis and 
testing 

- Critical Path 

- Adequate slack 

High risk items, work around 
Exit criteria for every milestone 

• Cost 

Defined cost for all milestones 

- Costs include NASA and contractor 

% Management and technical team (experienced) 
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Research and Development 

Degree of Difficulty (RD 3 ) 

R&D 3 

I A very low degree of difficulty is anticipated in achieving research and 
development objectives for this technology. 

Probability of Success in “Normal” R&D Effort > 99% 

II A moderate degree of difficulty should be anticipated in achieving R&D 
objectives for this technology. 

Probability of Success in “Normal” R&D Effort > 90% 

III A high degree of difficulty anticipated in achieving R&D objectives for this 
technology. 

Probability of Success in “Normal" R&D Effort > 80% 

IV A very high degree of difficulty anticipated in achieving R&D objectives for this 
technology. 

Probability of Success in “Normal” R&D Effort > 50% 

V The degree of difficulty anticipated in achieving R&D objectives for this 
technology is so high that a fundamental breakthrough is required. 

Probability of Success in “Normal" R&D Effort > 20% 


Measures of Programmatic Failure 

• Development difficulty 

- Technology Readiness Level Gap (Initial to TRL6) 

- Research and Development Degree of Difficulty 

- TPM gap 

• Requirements, requirements flowdown, interface requirements, etc. 

• Schedule 

- Defined schedule showing maturity increasing/adequate analysis and 
testing 

- Critical Path 

- Adequate slack 

- High risk items, work around 

- Exit criteria for every milestone 

• Cost 

- Defined cost for all milestones 

- Costs include NASA and contractor 

• Management and technical team (experienced) 
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NASA Program Schedule Actuals 


MER 

Gemini - Manned 
Skylab Workshop - Manned 
Mars Global Surveyor 
Pathfinder 
Centaur-G' - Launch Vehicle 
Voyager - Unmanned 
Viking Lander - Planetary 
Magellan - Planetary 
Viking Orbiter - Unmanned 
Apollo LM - Manned 
S-IVB - Launch Vehicle 
Apollo CSM - Manned 
Mars Observer - Unmanned 
Skylab Airlock - Manned 
S-ll - Launch Vehicle 
External Tank 
Shuttle Orbiter - Manned 
Spacelab - Manned 




□ ADP to PDR 
■ PDR to CDR 

□ CDR to Launch 


20 40 60 80 100 120 140 

Calendar Months 
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Life Cycle Milestones 
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- Phase E 
Operations 
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AOA Analysis ot Alternatives 

ASOR Annual Systems Operations Review 

CDR Critical Desion Review 

DR Decommissions Review 

FCA Functional Configuration Audit 

FOC Fuu. Operational Capability 

FOR Formal Qualification Review 

FRR Flight Readiness Review 

ICF Inoepehpent Cost Review 


IOC Initial Operational Capability SAR 

MCR Mission Concept Review SDR 

MSR Mission Design Review SFR 

WAR Non-Advocate Review SR 

OAR Operational Acceptance Review SRR 

ORR Operational Requirements Review SYR 

PCA Physical Conficuraion Aodit TRR 

PDR Preliminary Design Review 

PRR Production Readiness Review 


System Acceptance Review 
System Design Review 
System Functional Review 
Safety Review 

System Reouirements Review 
System Verification Review 
Test Readiness Review 



Development difficulty 

- Technology Readiness Level Gap (Initial to TRL6) 

- Research and Development Degree of Difficulty 

- TPM gap 

Requirements, requirements flowdown, interface requirements, 
etc. 

Schedule 

- Defined schedule showing maturity increasing/adequate analysis 
and testing 

- Critical Path 

- Adequate slack 

- High risk items, work around 

- Exit criteria for every milestone 

Cost 


- Defined cost for all milestones 

- Basis of costs (FTEs, facilities, hardware, etc.) 

• Management and technical team (experienced) 
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■ Many cupons tested 
1 Feeds sector test prog 

• Continues during sector test prog 

1 Used for sector design refinement 

• Essentially complete by FY95 
GE/NASA 


Combines components for integrated evals 

3 configurations tested 

Primary feed to annular test program design 

Secondary feed to core combustor test program design 

Uses non EPM materials 

GE/NASA 


3 generation tests of progressively complex design 
Gen I tests and Gen II design from separate contract 
P&W test feed annular rig test program design 
NASA test feed core combustor test program 
Uses non EPM materials 
P&W/NASA 


Applies to RQL configuration 

P&W/NASA participation 

Feeds annular rig test program design 


Same as 1. 0.2.6 
P&W participation 


Added shape fidelity over rectangular evals 
Two test series of angle configuration 
Feed core combustor test program design 
GE 


Feed products to test programs as developed 
NASA 


Evaluation of rectangular sector configurations 
Primary feed to annular test program design 


Low NOx Combustor 


1 -Pager Work Schedule 


Feed products to test programs as developed 
NASA 


Feed products to test programs as developed 
Universities 
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Minimal Technology Data Sheet 
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Assessing Technology Risk Using AHR 
(Analytical Hierarchical Process) 

• The AHP is based on the hierarchical decomposition of the 
prioritization or forecasting criteria down to the level at 
which the decision or forecast alternatives can be pair- 
wise compared for relative strength against the criteria. 

• The pair-wise comparisons are made by the participating 
experts and translated onto a numerical ratio scale. 

• The AHP mathematical model then uses the input pair-wise 
comparisons data to compute priorities or forecast 
distributions as appropriate. 


Analytical Hierarchical Process 


Individual Assessment 


Metric Interval 

Most Likely 

Relative Likelihood 

20 to 25 Units 

o 

5% 

As likely as 
35 to 40 



25 to 30 

o 

25% 

As likely as 
35 to 40 



30 to 35 

o 

75% 

As likely as 
35 to 40 



35 to 40 

• 

100% 

Most likely 
interval 




45 to 50 

o 

10% 

As likely as 
35 to 40 





Integrated Group Assessment 
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echnology Risk Assessment - Phase 3 
Summary Of Airframe Risk Assessments 


TA 

TECHNOLOGY PROJECT 

2 

STRUCTURAL HEALTH MONITORING - NORTHROP GRUMMAN 

2 

METALLIC CRYOTANK - BOEING 

2 

CERAMIC MATRIX HOT STRUCTURES - MRD 

2 

DURABLE ACREAGE CERAMIC TPS - BOEING 

2 

DURABLE ACREAGE METALLIC TPS - OCEANEERING 

2 

INTEGRATED AERO-THERMAL & STRUCTURAL THERMAL 
ANALYSIS - NASA 

2 

STRUCTURAL & MATERIALS/TANK/TPS INTEGRATION - NASA 

2 

STAGE SEP & ASCENT AERO-THERMODYNAMICS - NASA 

2 

MATERIALS & ADVANCED MANUFACTURING: PERMEABILITY 
RESISTANCE - NASA 

2 

LIGHTWEIGHT INFORMED MICRO-METEOROID RESISTANT 
TPS - NASA 

2 

ULTRA HIGH TEMPERATURE SHARP EDGE TPS - LMC 

2 

CERAMIC MATRIX COMPOSITE - SOUTHERN RESEARCH 


COST SCHED TECH 


As r VS! ■rvrixx 


I 

I 




Technology Risk Assessment - Phase 3 
Structural Health Monitoring (Shm 


TA-2 Airframe Northrop Grumman 

MAJOR RISKS 

o Cost - Cost of 8,000 sensors for full scale SHM could be very high, but is 
understood. 

Schedule - Critical schedule issue is availability of Composite Cryo-tank for testing. 
SHM starting at TRL 4 in 2002. No development issues affecting schedule. 

CD Technical 

> Reliability - Integration of 8,000 sensors into one reliable SHM is a risk 

> Testability - Availability of Full Scale Composite Cryo-tank for testing to achieve 
TRL 6 

CONTINGENCY PLAN SUGGESTION 

Use a subscale tank ( 1 8 to 20 ft diameter) to test SHM system 

NOTE: Only new or updated comments are contained in this report. Refer to Phase 2 
report for complete evaluation. No significant change in evaluation from Phase 2. 


Show Stopper - Lack of Funding for Composite Cryo-tank for 

Testing 

NOTICE: This information is technical data within the definition of the International Traffic in Arms regulation ( ITAR I and, or Export Control Administration Regulations ( EARl and is subject to the 
export control laws of the United States. Transfer of this data by any means to unauthorized persons, as defined by these laws, whether in the U. S. or abroad, w ithout an export license or other approval 
from the U. S. Department of State is expressly prohibited. 


25 


































Structural Health Monitoring (Northrop Grumman) 

Development Schedule 



1: They should meet this goal based on present information. 


2: NGC is starting with the SHM technology at a TRL level of 4 in 2002. They have plans to develop a structural 
health monitoring system and integrate it into a full-scale composite cryotank and complete test in 2005 
timeframe. So the critical element of this is really having available a full-scale composite tank with this system 
integrated into it in 2005. That's the biggest concern because the funding level could get cut on the full-scale 
development of a composite tank that is in a separate technology development/funding under GEN2. So, there 
are no major issues with respect to developing the SHM system that NGC is proposing here. The issue is with 
respect to the availability of a full-scale composite cryotank in 2005/2006 which could face some serious 
funding issues given that GEN2 is probably not going to carry two tanks to TRL = 6 (metallic and composite). 


5: If funding is maintained for the duration of the project, it is probable that it will come in on schedule. 


7: There is a trade-off that should be made between the amount of health monitonng and robustness of 
design/analysis. As the vehicle is used for repeated flights some of the health monitoring sensors will become 
inoperable and others will produce data that has increasing errors. At some point a decision will need to be 
made relative to how many flights can be achieved before the health monitoring system itself must be inspected 
and checked out for adequate performance. The cost of maintaining the health monitoring system should be 
weighed against the cost of increasing the robustness of design thereby reducing the need for health 
monitoring. The reliability of the health monitonng system must consider the sensors, the data system and 
everything that is needed to transfer the data from the sensor to the data system. The lowest reliability part of 
the system may be the vehicle installed data transmission lines (quite a nest of lines) which must pass through 
the vehicle requiring compromises to be made in other disciplines of the vehicle design. 


Goal: 2006 years 


Technology Success Data 


Technology Area: Airframe Technologies 

Technology Development: Composite Cryotank 



External Inspection Interval 
Flight Mission Life 
Internal Inspection Interval 
Leak Rate 
Operating Pressure 
Reliability 
Weight/Volume 

‘Weighted Technical Success: 
3 Combined Weighted Success: 


Assumption: The Low to High range contains 
100% of the possible v alues of the metric. 


Expected Value - Mean or 
average value of the 
estimated probability 
distribution. It is the value 
of the metric expected by 
the evaluators 


Expected Value Deviation - 
Deviation of the EV from the 
goal, calculated as follows: 

Absolute Value: EV - Goal 


Goal 

A minus sign in front of the 
calculated value indicates that 
the EV is worse than the goal. 


’ EV Deviation show by how much the EV misses the goal. It is omitted for certain metrics. 

Weighted Success is the average success probability of the metrics. 

3 Combined Weighted Success is average of technical and programmatic Weighted Success. 


0% - 20% 20%-50% 50%-100% 
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Delta Isp, 



IVfetalizBd 


M/anoed IVbterials 


Chanrber Ressire 


Oarbustion BTici 


Nozzle Efficiency 


OF Ratio 



3 

200 

0 . 07 : 

3 



10 150 


8 100 


0 . 


0 . 


0 . 


0 . 


0.031 


What is the your investment order? 
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Weighted Technology Impact Ranking 

(Quantitative assessment after tech portfolio selected and funded) 


Safety (45%) 

Loss of Crew 
Loss of Vehicle 
Loss of Mission 
Loss of Payload 
$/lb (35%) 

Launch Availability 
DDT&E - Average 
1st Unit Prod. Cos 
Annual Ops Cost ( 
Facilities Cost (10 
Technical (20%) 

Vehicle Empty Wei 
Vehicle GLOW 
Total Weighted Score 


V) 

C 

< 1 ) 

E 

Q) 


O 

on 




0.18 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.14 0.10 0.07 0.07 



3 itl .T5 


ESI 

ESI 

KE1 


■flKI 

TBI 

TIBI 

TIBI 

TBI 

TBI 

TIBI 

■flEl 

TBI 

TBI 

TBI 

TBI 







■HE1 

MflEI 

■TIBI 

TIBI 

TIBI 

TIBI 

TIBI 

TBI 

■tin 

TBI 

TIBI 

TIBI 

TBI 

TBI 



ESI 

ESI 

m 

■ilEl 

■Bel 

TBI 

TIBI 

TIBI 

TIBI 

TIBI 

TBI 

■«TE1 

TIBI 

TBI 

TBI 

iiBI 

TIBI 

ESI 

wbi 

IESI 

ESI 

KBI 

eh 

■rei 

TIBI 

TIBI 

TBI 

TIBI 

Tin 

TBI 

E E 

TBI 

TIBI 

TBI 

TBI 

TIBI 

ESI 

ES 


0.11 0.11 0.11 0.11 0.08 0.06 0.02 0.11 0.06 0.02 


0.06 0.06 0.03 0.06 0.06 0.06 0.00 0.00 


TBI TBI 

TBI 


■ 0.3 

0.3 

0.3 

TBI 

TBI 

o.3|a 

^1 0.3 

0.3 

0.3 


0.50 0.41 0.38 0.36 0.36 0.34 0.30 0.30 0.30 0.25 0.26 0.22 0.27 0.13 0.09 


Impact Assessment 


High 


Medium 


Low 



Comments on Investment Strategy 
and Impact Assessment Method 


• Very poor choice of technology portfolio (~two-thirds of 
technologies have low or negative impact) 

• Wrong requirements were developed 

• Systems analysis did not model the technologies 
correctly 





Impact on Requirements 
(weighted value functions) 








Technology Agency Impact Model 


Requirements 

Flowdown 



Technology Needs 


Enterprise 
Strategic 1 

1 

f i 

Missions / ^ 

Program 

3 

r i 

Architecture 

5 

i i 

Capability 

1 

i 

Technology [ 


Priority of missions within an Enterprise 


Percentage of total missions that architectures are utilized 


Percentage of proposed architectures that capability impacts 


Indexed technology impact on capabilities computed by systems 
analysis (not yet available for all Architectures) or by expert 
opinion 


Technology _ Capability . Architecture * Mission * Enterprise 
Impact Impact Impact Impact Impact 


Summary 

Technology Risk Assessment 

• Technology risk is based on the probability of technology 
development success versus the impact of the technology on 
the system 

• Technology development probability of failure is similar to any 
project. Should have defined WBS, requirements, schedule, 
cost, etc. 

• Expert opinion is used for assessment; AHP is one method to 
obtain and integrate the opinions. 

• Expert opinion or systems analysis can be used to define the 
impact of the technology on the system. 

• For total Agency impact, future enterprise missions need to be 
prioritized to assess technology global impact and risk. 
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